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ABSTRACT 


Gamification frameworks can aid in gamification planning 
for education. Most frameworks, however, do not provide 
ways to select, relate or recommend how to use game ele- 
ments, to gamify a certain educational task. Instead, most 
provide a “one-size-fits-all” approach covering all learners, 
without considering different user characteristics, such as 
gender. Therefore, this work aims to adopt a data-driven 
approach to provide a set of game element recommenda- 
tions, based on user preferences, that could be used by teach- 
ers and instructors to gamify learning activities. We anal- 
ysed data from a novel survey of 733 people (male=569 
and female=164), collecting information about user prefer- 
ences regarding game elements. Our results suggest that 
the most important rules were based on four (out of nine- 
teen) types of game elements: Objectives, Levels, Progress 
and Choice. From the perspective of user gender, for the fe- 
male sample, the most interesting rule associated Objectives 
with Progress, Badges and Information (confidence=0.97), 
whilst the most interesting rule for the male sample associ- 
ated also Objectives with Progress, Renovation and Choice 
(confidence=0.94). These rules and our descriptive analy- 
sis provides recommendations on how game elements can be 
used in educational scenarios. 


1. INTRODUCTION 


Gamification has been a widely popular phenomenon in the 
past few years, being used in various domains, including that 
of education. Gamification is defined as the use of game el- 
ements outside their scope (i.e., games or game playing) 
[1, 2]. However, educators often may not be familiar with 
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specific game-related concepts, or know how to use game 
elements, or may not have the resources or time necessary 
[3, 4, 5]. A solution is to employ conceptual gamification 
frameworks [6]. Still, existing frameworks lack resources and 
explanations on how to use game elements appropriately [5], 
especially when considering user preferences affected by de- 
mographic differences. Understanding users’ characteristics, 
such as gender, may be especially beneficial, e.g., in STEM 
education, where the well-known problem of ’the leaking 
STEM pipeline’! occurs [7]. 


In this paper, we apply a data-driven approach to provide 
insights into the educational domain, via the research ques- 
tion: “How can gender differences in preferences about gam- 
ification elements be used to support gamification design?” 
We conducted a very large survey (808 raw answers) allow- 
ing respondents to rank gamification elements. We based 
these elements on the works of Dignan [8] and Toda et al. 
[5], due to (a) the relatively large number and variety of 
elements, (b) the availability of synonyms used. Next, we 
used an unsupervised learning algorithm to generate Asso- 
ciation Rules to find patterns within the dataset, in order 
to understand relations among these elements, based on the 
users’ genders. Our main contributions are: (a) a survey” 
for extracting preferences for gamification for education, ap- 
plied to a large, varied number of respondents; (b) extract- 
ing gamification elements relevant to the different genders, 
for the educational domain; (c) extracting relations between 
these elements, relevant to the different genders; (d) insights 
into users’ acceptance of specific game elements, or groups 
thereof (and their relations). 


2. RELATED WORKS 


As there are very few frameworks focusing on gamification in 
education domains, we discuss: (i) existing models related 
to game elements, (ii) gamification studies on user charac- 
teristics, (iii) planning of gamification. 


‘dropout in STEM education 
"https: //forms. gle/hFgTT7kCqBKLqiPd8 
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Yee and Marczewski both proposed models on how to use 
game elements,using also large data collections [9, 10]; how- 
ever, their focus is different: they collected (a) players’ moti- 
vations towards online RPGs and (b) generic gamified appli- 
cations. [10] only provides recommendations of elements for 
behavioural profiles, but not user demographic characteris- 
tics, such as gender. Yee’s model additionally analysed be- 
haviours from a gender perspective, but only fo online RPG 
(World of Warcraft) players exclusively. A recent study by 
Shi and Cristea [11, 12] proposed a model and a set of rec- 
ommendations based on the Self-Determination Theory [13]. 
Their Motivational Gamification Strategies related game el- 
ements with each construct of the SDT, i.e., Autonomy, 
Competence and Relatedness, and implemented them [11], 
achieving positive results for each construct. Such studies 
show how motivational theories and gamification constructs 
can be related, as well support gamification in education. 
They however do not support the design process of gamifi- 
cation for educators and teachers. 


Denden et al. [14] conducted an experiment analysing user 
preferences (N = 120) over eight game elements within a 
gamified educational system, based on personality traits (the 
famous ’Big Five’ [15]). According to the authors, only 
extraversion, openness and conscientiousness affected stu- 
dents’ preferences for particular game elements. The au- 
thors also stated the importance of this kind of recommen- 
dation to designers and instructors when gamifying their 
learning environments. However, the gamification in edu- 
cation literature lacks studies which relate the acceptance 
and influence of game elements with users’ genders, involv- 
ing large-scale data [16]. One recent study [17] conducted 
an experimental study aiming at identifying differences be- 
tween male and female users (N = 70) towards ’gaming the 
system’ behaviours. It was shown that game elements led to 
male users decreasing their undesired behaviours; moreover, 
female users felt less competent than male users. Never- 
theless, although the results are interesting, the number of 
students who were analysed is still relatively small, with stu- 
dents from within a course context - whereas our study has 
a wider scale and variety of participants. 


Toda et al. [5] proposes a framework for blended class- 
room environments using social networks, via a list of rec- 
ommendations (names of gamified strategies) based on pre- 
vious studies in gamification in education. They also ap- 
ply Dignan’s game elements classification [8]. However, the 
gamified strategies proposed are solely based on literature. 
Nevertheless, their positive results show that game elements 
are suited for educational environments (e.g., classroom and 
digital platforms). As noted, other gamification frameworks 
focused on specific domains (e.g. Computational Thinking 
[18]). Klock et al [19]’s framework is usable for adaptive sys- 
tem. Still, Mora et al [6] note that this framework focuses 
on the researchers, rather than the stakeholders (teachers 
and instructors) and presents limited recommendations on 
game elements usage. 


Thus, whilst gamification shows potential benefits for ed- 
ucational applications, the gender differences in preference 
towards specific game elements needed further, large-scale, 
systematic studies, to better provide support for Data-Driven 
Gamification Design, as tackled by our current paper. 


3. DATASET AND METHODS 


Our survey on game elements contains 29 questions. The 
first part collected demographic information (age, favourite 
game setting, and gender). The second part asked to what 
extend certain game elements were relevant to users in the 
gamified educational system context, through a Likert Scale, 
from 1 “I think this element is irrelevant to me” to 5 “I think 
this element is highly relevant to me”. The game elements 
used and their advantages and potential drawbacks are pre- 
sented in https://tinyurl.com/y44kqvn5 based on [8] and 
used in [5] in an educational domain. 


Additionally to theoretical motivations, we further validated 
the selected gamification elements with 4 specialists in gam- 
ification, who were also teachers, via an interview, verify- 
ing the specialists’ acceptance of the used elements, con- 
cepts, as well as questions cohesion. Finally, a pilot sur- 
vey with 18 people verified the time spent and the consis- 
tency of the questions, before launching the main survey 
https://goo.gl/forms/d0i5WosBcMVWvQAK2. We then re- 
cruited surveyees through social networks, forums and digi- 
tal environments used by people who play games. 


In total, we collected 808 raw answers. Further cleaning 
removed data from users who: (a) did not answer all ques- 
tions; (b) claimed not having played any digital games; (c) 
were of age<0 or age>90. Then, we analysed our population 
characteristics based on demographic data. As the normal- 
ity test showed a non-normal distribution, a Mann-Whitney 
test [20] was used to compare males and females. 


Finally, we used association rule mining to analyse the re- 
lations amongst our data, based on gender. Unsupervised 
learning was used as we do not have any predefined la- 
bels (outputs) and also to understand the relations between 
the elements (different from clustering which create groups 
based on all variables of the dataset).The algorithm analyses 
the items’ frequency (support) and renders a level of confi- 
dence, ranging from 0 to 1 (where 1 is the maximum confi- 
dence). The confidence can also be supported by conviction 
[21], lift and leverage -- both measuring the independence 
of items. 


4. RESULTS AND DISCUSSION 


4.1 Gender differences 

After filtering, we retrieved 733 valid answers (90.72%). We 
applied Cronbach’s a on the second group of questions (as 
game elements were based on a Likert scale) and achieved 
an a = 0.83 (high reliability factor [22]). Our sample is 
varied in terms of age (ranging from 13 to 68), but limited in 
terms of experience in playing (at least a year: by design and 
filtering) and country of origin (Brazil; due to convenience 
sampling). Nevertheless, the sampling size is much larger 
than the recommended one (733 >> 384; people playing 
online games estimated at 700 mio; confidence level 95%). 


We further organised our valid answers into two groups: 
males (N = 569) and females (N = 164), and verified the 
distribution of the data using the Shapiro-Wilk normality 
test. The result showed that our data rejected the null hy- 
pothesis (p < 0.05), so we adopted non-parametric tests in 
further analyses. Table 1 summarises the result. 
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Table 1: Relevance of game elements, averaged per 
gender 


Gender (mean) Mann-Whitney 


Element Female Male W p-value 
Point 3.76 3.68 44836 0.429 
Level 4.14 4.21 48418 0.427 
Cooperation 3.62 3.86 52306 0.013 
Competition 3.26 3.56 53016 0.006 
Renovation 4.16 3.78 35878  2.36e-03 
Progress 4.24 4.32 48856 0.312 
Objective 4.41 4.4 45902 0.791 
Puzzles 4.14 3.91 40636 0.008 
Novelty 4.05 4.16 49530 0.197 
Chances 3.68 3.61 44901 0.447 
Social Pressure 3.43 3.65 51142 0.05 
Acknowledgement 3.85 3.73 44673 0.387 
Data 4.05 4.09 46675 0.994 
Scarcity 3.16 3.42 52468 0.011 
Choice 4.07 4.23 50267 0.08 
Time Pressure 3.16 2.97 42711 0.09 
Economy 3.41 3.42 46738 0.973 
Sensation 3.62 3.1 37094 1.17e-02 
Classification 3.51 3.72 51340 0.042 


Table 1 shows many significant differences (p-value < 0.05). 
Interestingly, Competition, Cooperation, Social Pressure, 
Scarcity and Classification were considered slightly more rel- 
evant by the males, whilst Renovation, Puzzle and Sensation 
elements were considered more relevant by females. Time 
pressure was disliked by males, but not as much by females. 


4.2 Descriptive Analysis 

Comparing surveyees, for elements preferred in different pro- 

portions, which are statistically significant between males 

and females, Cooperation was more relevant to males (57.3%); 
with 41.6% males selecting highly relevant, vs. 31.1% fe- 

males (Table 2). 


Gender 


Bh Femate 
BB vac 


Figure 1: Favourite game elements for males (a,b- 
correspond to Tables 2,3, rsp.) 


A more drastic difference appeared when more than 25% 
of the females did not consider Competition to be relevant, 
versus 54.5% males. This result suggests that males may 
perceive social interactions, such as Competition elements 
and Cooperation, as highly relevant overall in games, with 
a slight preference of Competition. For females, however, 
Competition is not that relevant, but, surprisingly, Cooper- 


ation is only marginally relevant. Social Pressure is signifi- 
cantly less liked by females (difference of 7.7%). 


Table 2: Cooperation, Competition and Social Pres- 
sure answers. 


Cooperation Competition Social Pressure 
Sc F % M % F % M % F % M % 
, 12. Ft 6. 20 189) AD. TA dd 8.5 32 5.6 
2 20-122 $39.3 -25--15.2 80 — 14.1. 26- 1b 863 IT 
3 38 23.8 100 17.6 44 26.8 1387 241 45 27.4 146 25.7 
4 43 26.2 142 25 43 26.2 135 23.7 34 20.7 160 28.1 
5 51 31.1 2387 41.6 32 19.5 175 30.8 45 27.4 168 29.5 


Scarcity was instead favoured (significantly) by males (Ta- 
ble 3), where 37.8% of the females are indifferent. Classi- 
fication, also a social element, was considered significantly 
more relevant by males. 50.6% of the females considered it 
relevant, against 60.6% males (Table 3). Based solely on 
our descriptive analysis, we observed that the male popula- 
tion considered limited or rare tasks, allowing, e.g., rewards 
such as interaction or collecting titles, as relevant. Again, in 
practice, this information allows the teachers to create titles 
for completing specific tasks during their lectures e.g., by 
giving a title of Speedster’ to the student who completes a 
list of task correctly and quicker than the others. 


Table 3: Scarcity and Classification answers 
" Searcity —C Clsssification = 
Sc F % M % F % M % 
16698 32 56 13 79 25 44 
25 15.2 78 13.7 14 85 68 = 11.9 
62 37.8 189 33.2 54 32.9 131 23 
39 23.8 161 28.3 42 25.6 163 28.6 
22 13.4 109 #%19.2 41 25.0 182 32.0 


OU HY] Col DO} Fe 


As for the elements most favoured by females (Figure 2), 
Renovation scored highest. 76.2% said it was relevant, with 
50% considering it highly relevant. In contrast, only 30.0% 
of the males considered it highly relevant, with almost 30% 
indifferent (Table 4). In a learning context, this may tell 
the teacher that female students might be more pleased with 
features as “continue”, “try again” or be given ’extra lives’. 


Another element highly relevant to females was Puzzles: 
80.8%, against 68.1% of males; with 21.9% males indiffer- 
ent. Again, females, in this scope, considered that testing 
their skills was more relevant than males did. The Puzzle 
and Renovation elements, when combined in practice, allow 
problem solving, with the opportunity to correct mistakes. 


Finally, the Sensation element was considered more relevant 
by females. More than half (54.8%) of the female sample 
considered it relevant, against 42.5% of the males. This 
could be explained by Sensation being related to the user 
experience [8], and, based on Table 4, we can infer that the 
most relevant elements for the female sample were related 
to the experience, rather than social ones. This means that 
they may perceive tasks that involve their senses, e.g., with 
a visual or phonetic appeal, as more relevant, which could 
further be redone whenever they wish, to improve a certain 
skill through challenges. In practice, this means that using 
materials and resources that are more visual appealing may 
be more pleasant to female students than the male ones. 
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Figure 2: Favourite game elements for the female 
sample 


Table 4: Renovation, Puzzle and Sensation answers 


Renovation Puzzle Sensation 


Sc F % M % F % M % F % M % 


it 4 24 14 25 #5 3 17 3 12 73 108 19 


09 6.9 39 68 § 49 4) 7 21 128 97 18 


2 
3 26 15.8 167 29.3 20 12.2 125 22 Al 25 122) 21.4 
4 


43 26.2 161 28.3 57 348 180 316 34 20.7 115 20.2 


5 82 50 188 33 74 45.2 207 364 56 341 127 22.3 


4.3 Association Rules analysis 

To identify the strongest rules for each gender and to ver- 
ify how the rules found matched or complemented the find- 
ings from our descriptive analysis (Section 4.2), we used the 
Apriori algorithm in Weka, with: (i) minimum support of 
10% for the male sample size and 20% for female (to bal- 
ance sample sizes); (ii) minimum confidence of 90%; and, 
after applying those attributes, (iii) we used the measures 
of interest conviction, lift and leverage to find the most in- 
teresting rules [23]. Using this setting we found a total of 
11 rules for the female sample and 13 rules in the male one. 


The majority (>90%) of the rules were based on the Ob- 
jective element, which suggests its overall popularity. This 
translates into a general recommendation towards using ’Ob- 
jective’ elements in the educational gamification design, such 
as missions, milestones and quests, to guide the students. 
In this work we focused on analysing the most interesting 
rules in female and male samples. For females, the strongest 
rules were associated with the Objective (Table 5). The lift 
> 1 and leverage near 0 indicate that our items are inde- 
pendent and have a positive correlation, and the conviction 
between 1 and 5 indicates that these are interesting rules. 
The strongest rule relates Progress, Acknowledgement and 
Data elements (e.g., Representations of progression, badges 
and medals and results screen) with Objective (e.g., mis- 
sions and quests). Rules regarding Progress and Level were 
also amongst the 10 strongest. Thus, we can suggest that 
teachers and instructors should use Acknowledgement (such 
as badges and trophies), with other elements associated with 
the personal enhancement of users (Progress and Level). 


As for the male sample, Objective was also the main ele- 
ment but, in contrast to the females, we did not find any 
rules (with confidence > 90%) related to elements that were 
most relevant to the male population (Table 6). There was 
only 1 rule that specified a social element (Social Pressure) 


amongst all the 14 rules. We can observe that Progress ap- 
pears in almost all the rules, followed by Choice, appearing 
in seven rules. This means that, in our sample, designers 
and teachers should consider quests and missions that con- 
tain a form of progression and allow the students to make 
meaningful choices; those choices can be tied to a challenge 
(Rule 14), to transactions (Rule 24) and points (Rule 16). 


Based on the data on Tables 5 and 6, we can observe that 
Objective associated to Progress is a concept that is (gener- 
ally) well accepted by both genders. This means that teach- 
ers and designers should focus on, e.g., developing quests 
(which can be tied to their original learning objectives) that 
allow the learners to place themselves within the task. This 
is important, since in some educational context, students do 
not know why they are learning a specific content; and con- 
sequently, may become demotivated [24]. In practice, this 
means that teachers can create milestones or goals, allowing 
students to visualise their progress towards this goal. Thus, 
guidelines can be provided to teachers, to convert their ob- 
jectives in their classes into milestones or quests. Addition- 
ally, other representations of Progress, showing the users 
where they are in the course could be implemented, such as 
those supported by Levels, Points and Data. 


4.4 Further Discussion 

We consider this work to be important, as, with the advent 
of ’big data’, various theoretical assumptions and statements 
can now be backed up by (significant) evidence. In the case 
of game elements, there is firstly a vast (not always research- 
based) evidence that games are linked to motivation, and 
keep players ’in the flow’ [25]. Some studies even link spe- 
cific game elements to higher levels of commitment or mo- 
tivation [9]. Based on this evidence, as well as theories of 
motivation, gamification has been proposed for education. 
Currently, however, the data supporting these assumptions 
is scarce. There is a lot of small-scale empirical evidence, at 
classroom-scale, of approaches that showed mixed successes 
[26, 27]. In a similar way, there is evidence that gamification 
can also have undesirable effects [28]. This clearly points to 
the fact that there are parameters which need taken into 
consideration, which may influence the outcomes of gami- 
fied approaches to education. In this study, we specifically 
focus on demographic parameters - namely, gender. 


Gender in education has been brought to the fore recently, 
with the advent of initiatives such as the "Athena SWAN”? 
initiative towards gender equality in Higher Education in the 
UK, as well as similar initiatives world-wide. Importantly, 
equality doesn’t mean ’one size fits all’: on the contrary, gen- 
der equality means that the provision of education takes into 
account specific preferences that may be gender related. Ina 
similar vein, certain types of games appeal to certain demo- 
graphics and not others. For instance, card-related games 
are potentially more appealing to women, and first-player- 
shooter games to men (although, of course, preferences can 
vary) [29]. 


Further analysis of Table 1 shows that male and female pref- 
erences of some elements is relatively similar; e.g., Data, 
Economy, Objective are almost identical, and some are only 


https: //www.ecu.ac.uk/equality-charters/athena-swan/ 


441 Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019) 


Table 5: Relevant association rules for female sample 


ae If Then Conf Lift Lev Conv 

1 {progress, acknowledgement, data} {objective} 0.97 1.63 0.08 7.04 

2 {level, progress, acknowledgement} {objective} 0.97 1.62 0.08 6.84 

3 {progress, acknowledgement } {objective} 0.96 16 O01 6.04 

4 {level, acknowledgement} {objective} 0.95 1.59 0.09 5.5 

5 {point, acknowledgement } {objective} 0.95 1.58 0.08 4.96 

6 {progress, puzzles} {objective} 0.94 1.57 O01 4.73 

7 {puzzles, novelty} {objective} 0.92 1.54 0.07 3.82 

8 {novelty, acknowledgement } {objective} 0.92 1.54 0.07 3.72 

9 {acknowledgement, choice} {objective} 0.92 1.54 0.07 3.72 

10 {acknowledgement, data} {objective} 0.91 1.52 0.08 3.54 

11 {puzzles, acknowledgement} {objective} 0.9 1.51 0.08 3.38 

Table 6: Relevant rules to male sample 

ae If Then Conf Lift Lev Conv 
12 {renovation, progress, choice} {objective} 0.94 16 0.04 5.37 
13 {progress, social pressure, data} {objective} 0.93 1.59 0.04 5.04 
14 {progress, puzzles, acknowledgement } {objective} 0.93 1.59 0.05 5.16 
15 {level, renovation, progress} {objective} 0.93 1.59 0.04 4.96 
16 {point, objective, puzzles} {level} 0.92 1.93 0.05 5.74 
17 {level, progress, puzzles, choice} {objective} 0.92 1.57 0.04 4.27 
18 {progress, acknowledgement, data} {objective} 0.91 1.55 0.05 4.13 
19 {point, progress, choice} {objective} 0.91 1.55 0.04 4.01 
20 {progress, novelty, data, choice} {objective} 0.91 1.55 0.04 4.01 
21 {progress, novelty, acknowledgement, choice} {objective} 0.91 1.55 0.04 3.95 
22 {renovation, progress, novelty} {objective} 0.91 1.55 0.04 3.92 
23 {level, progress, data, choice} {objective} 0.91 1.55 0.04 3.92 
24 {progress, novelty, economy} {objective} 0.91 1.54 0.04 3.78 
25 {progress, choice, economy} {objective} 0.91 1.54 0.04 3.78 


slightly different. Thus, some game elements may be per- 
ceived similarly by males and females - which makes the 
teacher’s job much easier, in terms of design choices. This 
also puts more emphasis on the game elements where large 
differences exist, as well as on game elements where the dif- 
ferences in preference are slight, but statistically relevant. 


Some of the results obtained were surprising: for instance, 
we expected females to appreciate cooperation more than 
males, but results (see section 4.2) showed otherwise. We 
did, on the other hand, obtain the expected results in terms 
of preference for competition. It is possible that online social 
interaction overall is perceived differently by males and fe- 
males; for instance, females may perceive any type of social 
interaction online, where people are not known in advance, 
and anyone from anywhere can participate, as potentially 
threatening. These types of areas need further analysis. 


For educational applications, it may seem that such poten- 
tial ‘fears’ are less likely in controlled (classroom, or classroom- 
based) environments. However, for example, on Massive 
Online Open Courses (MOOCs), where people can partici- 
pate from anywhere, such issues can again prevail. In fact, 
research on social interactions on MOOCs (e.g., comments, 
etc.) shows a predominance of males performing such ac- 
tivities. In contrast, females preferred puzzles (which can 
be solved also as solo-player) and the Renovation’ element 
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(see Figure 2), which allow for an independent style of play 
where one focuses only on ones own progress, instead of be- 
ing interrupted by others. 


5. CONCLUSIONS 


This work presents our approach based on DDGD, towards 
planning of gamification in the educational information sys- 
tems domain by using data mining. The main contribution 
of our work is to present a poll of gamified strategies tied 
to male and female genders. Furthermore, we use real data 
to aid in the decision process of teachers and instructors 
is selecting gamification strategies. Through our data, we 
could identify that males would make more use of social in- 
teractions, with strong confidence rules pairing gamification 
elements Progression and Choice. For the females, we iden- 
tified that user experience and rewards are more relevant, 
with association rules indicating a strong confidence for the 
need of Acknowledgement and Progression. We believe that 
this work can impact the way teachers perceive and apply 
gamification in their environments, consequently improving 
students’ engagement and motivation through a game-like 
experience. 
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